Papers with contrastive decoding
Speculative Contrastive Decoding (2024.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. |
| Approach: | They propose a decoding approach that leverages predictions from smaller language models to achieve both decoding acceleration and quality improvement. |
| Outcome: | The proposed method achieves both decoding acceleration and quality improvement on four diverse language tasks. |
Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods analyze training data with member and non-member contexts, overlooking potential insights from both member and not-member. |
| Approach: | They propose a method that leverages asymmetric distributional shifts induced by member and non-member contexts through contrastive decoding to enhance membership inference. |
| Outcome: | The proposed approach outperforms the current state-of-the-art on the WikiMIA benchmark and is robust against various text manipulation techniques. |
Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models lack contextual knowledge, resulting in text with factual inconsistencies or contextually unfaithful content. |
| Approach: | They propose a method that integrates contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. |
| Outcome: | The proposed method improves context grounding during generation without training. |
Navigating the OverKill in Large Language Models (2024.acl-long)
Copied to clipboard
Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Xun Zhao, Dahua Lin
| Challenge: | Recent studies have highlighted a tendency among large language models to refuse to answer benign queries. |
| Approach: | They propose a model-agnostic approach to reduce excessive attention to harmful words like ‘kill’ and a method to decode the next-token predictions by contrastive decoding. |
| Outcome: | The proposed approach reduces the refusal rate by 20% while having little impact on safety. |
SH2: Self-Highlighted Hesitation Helps You Decode More Truthfully (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made great progress in text generation but suffer from hallucinations during reasoning and generation. |
| Approach: | They propose an inference-time method to help LLMs decode truthfully by selecting tokens with the lowest probabilities and concatenating them to the original context. |
| Outcome: | The proposed method improves LLaMA-7b, LLama2-7b and Mistral-7b on hallucination tasks. |
DVD: Dynamic Contrastive Decoding for Knowledge Amplification in Multi-Document Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate information with hallucinations due to uneven retrieval quality and irrelevant contents. |
| Approach: | They propose a decoding strategy which dynamically amplifies knowledge from selected documents during the generation phase. |
| Outcome: | The proposed method outperforms other decoding strategies on ALCE-ASQA, NQ, TQA and PopQA benchmarks. |
SCOTT: Self-Consistent Chain-of-Thought Distillation (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LMs) generate free-text rationales for their predictions via chain-of-thought prompting, but there is little guarantee that the generated rationale is consistent with LM’s predictions or faithfully justify the decisions. |
| Approach: | They propose a faithful knowledge distillation method to learn a small, self-consistent CoT model from a larger teacher model by contrastive decoding. |
| Outcome: | The proposed method yields comparable performance but is less faithful than baselines. |
Regularized Contrastive Decoding with Hard Negative Samples for LLM Hallucination Mitigation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are prone to generate hallucinations, which can undermine their reliability in high-stakes applications. |
| Approach: | They propose a method to capture hallucination signals for mitigating hallucis in large language models by regularizing the model's internal signals to a weaker model . |
| Outcome: | The proposed method achieves better hallucination mitigation performance on four benchmarks. |
HICD: Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often generate hallucinations, producing outputs that are contextually inaccurate or factually incorrect. |
| Approach: | They propose a method that selects attention heads crucial to the model's prediction as inducing heads and induces hallucinations by dispersing attention of these inducers. |
| Outcome: | The proposed method significantly improves performance on tasks requiring contextual faithfulness, reading comprehension, and question answering. |
Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models are not free from the issue of Object Hallucination (OH) OH is a phenomenon where LVLMs generate hallucinated objects and descriptions in their outputs. |
| Approach: | They propose a method to suppress OH by referencing images from AI-generated images at the logit level. |
| Outcome: | The proposed method significantly improves existing methods on visual contrast decoding. |
Alleviating Hallucinations of Large Language Models through Induced Hallucinations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models generate inaccurate or fabricated information, a phenomenon known as hallucinations. |
| Approach: | They propose a simple strategy to induce-then-contrast decode LLMs to enhance their factuality . they first induce hallucinations from the original model and penalize them . |
| Outcome: | The proposed strategy improves factuality of large language models across task formats, model sizes, and model families. |
Reimagining Safety Alignment with An Image (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to large language models face inefficiency, fragility, or architectural constraints, resulting in inefficient performance and heightened over-refusal in cross-modal tasks. |
| Approach: | They propose an optimization-driven visual prompt framework that enhances security and reduces over-refusal at the same time. |
| Outcome: | The proposed framework enhances security and reduces over-refusal while maintaining robust safety while reducing unnecessary denials. |
Achieving Stronger Generation via Simple Contrastive Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years have witnessed remarkable progress in large language models (LLMs). |
| Approach: | They propose a framework for contrastive decoding to enhance instruction-tuned models. |
| Outcome: | The proposed framework improves model performance without additional data or computational resources. |
Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that RALMs generate baseless information or contradicts with the retrieved context. |
| Approach: | They propose a lightweight monitor that leverages fine-grained decoding dynamics to synchronously detect unfaithful sentences. |
| Outcome: | Empirical results show that SynCheck outperforms baseline faithfulness detection and FOD outperformed traditional strategies in terms of faithfulness. |
Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used as evaluators but the reliability of the outcomes remains a challenge. |
| Approach: | They show that LLM judge outputs are highly sensitive to pre-defined score ranges and that similar biases exist among models from the same family. |
| Outcome: | The contrastive decoding of LLM judge outputs achieves 11.7% relative improvement in Spearman correlation with human judgments, averaged across score ranges. |
Contrastive Decoding: Open-ended Text Generation as Optimization (2023.acl-long)
Copied to clipboard
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis
| Challenge: | Using a language model, maximum probability is a poor decoding objective because it produces short and repetitive text. |
| Approach: | They propose a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. |
| Outcome: | The proposed approach outperforms four strong decoding algorithms in automatic and human evaluations across wikipedia, news and story domains. |
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training-free methods are vulnerable to the attention sink phenomenon . Existing methods include contrastive decoding and auxiliary expert models . |
| Approach: | They propose a training-free attention intervention that constructs a PAD map to identify semantically core visual regions and applies per-head Median Absolute Deviation Scaling to adaptively control the intervention strength. |
| Outcome: | The proposed intervention improves visual grounding and reduces hallucinations on multiple LVLMs and benchmarks. |
Securing Multi-turn Conversational Language Models From Distributed Backdoor Attacks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have acquired the ability to handle longer context lengths and understand nuances in text, expanding their dialogue capabilities beyond a single utterance. |
| Approach: | They propose a decoding time defense that scales linearly with the input sequence length and reduces the backdoor to as low as 0.35%. |
| Outcome: | The proposed framework is generalizable, compatible with any trigger in an adversary’s toolbox in a plug-and-play manner. |
EpiCoDe: Boosting Model Performance Beyond Training with Extrapolation and Contrastive Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance performance of Large language models are limited due to the cost of training data and privacy concerns. |
| Approach: | They propose a method that enhances a finetuned model with its inferior version and adopts contrastive decoding to reduce predicted errors. |
| Outcome: | The proposed method outperforms existing methods in data-scarcity scenarios across three domains and shows that it is more robust and robust. |
Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack explicit alignment between source and target contexts, leading to unfaithful translations. |
| Approach: | They propose three learning strategies to encourage LLMs to pay more attention to source context . they use a dataset to test the effectiveness of their model across multiple language pairs . |
| Outcome: | The proposed model reduces hallucinatory translation and improves fidelity across multiple languages. |
Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization (2025.acl-long)
Copied to clipboard
Lei Huang, Xiaocheng Feng, Weitao Ma, Yuchun Fan, Xiachong Feng, Yangfan Ye, Weihong Zhong, Yuxuan Gu, Baoxin Wang, Dayong Wu, Guoping Hu, Bing Qin
| Challenge: | Existing frameworks for retrieval-augmented large language models (LLMs) are lacking in LFQA faithfulness testing. |
| Approach: | They propose a framework to teach retrieval-augmented large language models to explicitly discriminate between faithful and unfaithful generations. |
| Outcome: | The proposed framework outperforms GPT-4o in LFQA scenarios and outperformed existing benchmarks. |
EAIR: Entity-aware Inference-Time Knowledge Routing for Multi-Hop Knowledge Editing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing in-context knowledge editing methods suffer from paraphrase sensitivity . Existing methods interfere with the model's original knowledge and can degrade its inherent capabilities. |
| Approach: | They propose a novel entity-aware inference-time knowledge routing method to address paraphrase sensitivity. |
| Outcome: | The proposed method achieves highest strict case accuracy in 11 of 12 settings, significantly reducing paraphrase sensitivity. |
Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to decode text to the most probable sequence have been proposed to address these challenges by improving coherence, diversity, and resemblance to human-generated text. |
| Approach: | They propose a novel decoding strategy that extends contrastive search by incorporating an adaptive degeneration penalty informed by the model’s estimated uncertainty at each generation step. |
| Outcome: | The proposed approach improves creativity and coherence while maintaining coherency across model architectures, languages, and datasets. |
MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LVLMs have shown remarkable performance in visual-language understanding for downstream multimodal tasks. |
| Approach: | They propose a method to alleviate hallucinations by masking the “image heads” in LVLMs . |
| Outcome: | The proposed method alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs. |
PruneCD: Contrasting Pruned Self Model to Improve Decoding Factuality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to decode early exit logits in large language models are lacking in factuality and accuracy. |
| Approach: | They propose a contrastive decoding method that constructs the amateur model via layer pruning rather than early exit. |
| Outcome: | The proposed method improves factuality with minimal inference overhead and is robust and practical. |
DiVE: Decoupling Intra-layer Visual Evidence for Mitigating Hallucinations in Large Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing decoding-based approaches do not explicitly decouple visual evidence from mixed vision–language representations. |
| Approach: | They propose to decouple visual evidence from mixed vision–language representations by dynamically identifying layers enriched with visual information and performing intra-layer decoupling to extract aggregated visual evidence. |
| Outcome: | Experiments show that DiVE achieves state-of-the-art performance on multiple benchmarks. |